Widen dequantized product to 64 bits in highbd quantize helpers

[edcote]

• Multiplying the quantized coefficient magnitude by the dequantizer step in 32-bit signed arithmetic before passing the product to the 64-bit rounding macro can overflow signed 32-bit integers.
• Casting the quantized coefficient magnitude to the 64-bit intermediate transform type before multiplication prevents signed integer overflow and matches the floating-point and trellis quantizer paths.