Assume the position is the top left corner. Assume the widest or tallest an object can be is 256 pixels. Assume that sprite offsets are always positive. (If you need negative offsets, add the maximum negative offset before drawing, and subtract it after drawing. Then offset your offset data accordingly so it can be exclusively positive.)
Subtract the camera position from the object's position and store the low byte. (The high byte is in A. I guess store this too if you don't want a lot of duplicated similar code.)
If the high byte is zero, add the width of the metasprite to the low byte. If the carry is clear, individual clipping is not needed for the sprites in that dimension. It is fully on screen for that dimension. Otherwise it is partially off screen on the right.
If the high byte is 255, add the width of the metasprite to the low byte. If the carry is set, the object is partially on screen on the left, otherwise it is fully offscreen. Return and do nothing if fully offscreen.
If the high byte was anything but 0 or 255, the object is fully offscreen. Return and do nothing.
The high byte check could look like this:
Code:
lda OBJxlow,x
sec
sbc scrollxlow
sta templow
lda OBJxhigh,x
sbc scrollxhigh
beq partiallyonscreenleft
cmp #255
bne fullyoffscreen
lda templow
clc
adc OBJwidth,x
bcc fullyoffscreen
;If here, object is partially on screen on the right
partiallyonscreenleft:
lda templow
clc
adc OBJwidth,x
bcc fullyonscreenx
;If here, object is partially on screen on the left
fullyonscreenx:
;If here, the object is fully on screen in the X dimension
fullyoffscreen:
rts
If you know the object is fully onscreen, you can just add and store your offsets.
If you know the object is partially on screen with a high byte of zero, you add the low byte to the offset, and branch on carry set to not drawing the sprite.
If you know the object is partially on screen with a high byte of 255, you add the low byte to the offset, and branch on carry clear to not drawing the sprite.
So: You use the high byte of the initial 16bit subtract to find out if the object is fully on screen, fully offscreen, partially off screen on the right, or partially off screen on the left.
If fully offscreen, you don't even need to check the other dimension. No drawing should happen.
If fully on screen, you only need to add the low byte to the offset and store it. No high byte check at all.
If partially off screen on the left, you need to add the low byte to the offset but only store if the carry is set.
If partially on screen on the right, you need to add the low byte to the offset but only store if the carry is clear.
Obviously this requires multiple drawing algorithms for the fastest possible result. I had two. One for fully on screen (the high byte in both dimensions was zero, and there was no carry set after adding the width for either) and one for partially on screen. (Which meant I had to do the 16 bit add even if one of the dimensions was fully on screen.) I figured that was fine.
Edit: If you don't have duplicate routines for each type of partial offscreen, you still have to do a 16bit add for every sprite, but you just do a non zero branch.
Code:
lda [reserved4],y;X Offset
clc
adc <reserved0;X Position
sta OAM+3,x
lda <reserved1
adc #$00
bne dms.partial.skipsprite.twoiny
Edit2: Knowing the direction of partially offscreen saves you the load of the high byte and the add. (You just need to branch depending on carry from the low byte.)
Fully on screen is just this:
Code:
;35 bytes
iny
lda [reserved4],y;Y position
clc
adc <reserved2
sta OAM,x
iny
lda [reserved4],y;X Position
;clc
adc <reserved0
sta OAM+3,x
iny
lda [reserved4],y
;clc
adc <reserved7;This should guarantee a clear carry
sta OAM+1,x
iny
lda [reserved4],y
sta OAM+2,x
txa
;clc;guaranteed clear above
adc #4
tax;Carry not guaranteed anything after that add since oampos wraps
Which I also unrolled.