As pointed out, you can not directly connect two hosts to a device, so if you simply connect them and someone has put it on both ends, you'd have a problem.
However, there is also a problem if only one end is connected. USB, in particular the high-speed mode (480 Mbps) is controlled as impedance. If you connect both data lines of the connectors you end up with what is known as a stub in high-frequency design. The cable to the unused connector will reduce the performance of the active connector.
To do this correctly, you want an IC multiplexer. You can purchase dedicated USB2.0 multiplexers designed specifically for this type of application, such as the TS3USB30. This would allow you to connect the data lines from both ports to the mux inputs and connect the output to the device internally. The mux will disconnect the unused connector that will disconnect the transmission line stubs.
For power, I would probably use a power multiplexer like an OR-diode circuit. The VBUS line from one of the ports (before the power multiplexer) can then be used as input for the data multiplexer.
Grounding would be common (connected) between the two USB ports and the device.