Roadmap for official rectangular training and inference support?

[qsq3]

Hi RF-DETR team,

Is full rectangular training/inference officially supported, or still experimental? If it is still in progress, is there any ETA or recommended workaround for 16:9 datasets?

[isaacrob]

We don't have a specific plan to work on this for the time being. We recommend trying your dataset squashed to square and seeing how the accuracy works out. The model was pretrained to be able to handle distortions of that flavor.

[aholliday-kr]

I'd like to second this as an issue. I began by squashing my dataset to square as @isaacrob recommended. I then did a quick hacky change to make the code resize things to a 3:4 aspect ratio (the native ratio of my dataset) instead of to a square, trained a second model this way, and compared the results. Using the native aspect ratio gave a considerable improvement in accuracy, around 20%. So there are benefits to not having to distort non-square input images. It would be great if I could do this without using my now somewhat out-of-date copy of the RF-DETR codebase with my hacky fix.

Incidentally, this might also be a reason to provide a DINOv3 backbone option, since as I understand it, DINOv3 models are more flexible about the input image shapes they will accept.

[isaacrob]

@aholliday-kr are you finetuning the pretrained model?

[aholliday-kr]

@isaacrob Yes, I am.

[isaacrob]

Got it. Could you link the change somewhere?

[aholliday-kr]

@isaacrob You can see my fork here: https://github.com/kinsolresearch/rf-detr
Note that I also added code to pad images that are the wrong aspect ratio instead of stretching them to the required ratio, since my dataset includes a minority of images that aren't 3:4. This change was made later on, after the square-vs-3:4 experiment I described above.

[qsq3]

@aholliday-kr Thanks

[isaacrob]

cool! looked at your changes. so to be clear, you would want that at train time we allow you to specify an aspect ratio that gets enforced at train and test?

[aholliday-kr]

@isaacrob In a perfect world I wouldn't need to specify an aspect ratio at all: images would be handled at whatever their native asprect ratios were. But specifying an aspect ratio at train time as you describe would certainly be an improvement. It would also be nice to be able to control whether the specified ratio is to be enforced by stretching or by padding.

[isaacrob]

Got it. ViT models don't naturally handle variable aspect ratio, there's a lot of work on it and my favorite technique is NaViT but unfortunately that training technique isn't directly applicable to a DETR. Because of that, we kept our models square during pretraining. Other people have found significant decay when using non square aspect ratios, and while it seems to be working in your case I'm not really convinced that it'll work generally, especially without significant data.

Having said that, it might be good to add the option. Just don't want to cause people to end up shooting themselves in the foot by accident!

Would you mind saying a bit more about your use case? How much data did you use, triple check that you used the pretrained weights, any other things worth noting about how you trained it that differ from just doing model.train?