I can't apply a line load or a member load to a truss where the chords have a cross section as shown in the attached picture. The applied load is only acting in the z-direction. If I try to run a load combination (second order) I'll get a "stiffness matrix is singular" error.
If I run a stability analysis using the "Calculate without loading for instability check... " the mode shape shows that the lower chord wants to deflect in the lateral direction.
Do I need to apply some lateral support using a shear panel? But this would only have effect on the upper chord.
You’ve already correctly identified the core issue. Your truss beam is supported only at the top chord at both ends, which provides very little resistance against torsion. As a result, the truss is susceptible to lateral-torsional buckling, even under purely vertical loads.
Your approach to calculate the unstable mode shape was excellent. From this, you can directly identify the critical stability problem.
To increase torsional resistance or improve the lateral stability of the truss, considering a shear panel can be very effective. Even if the shear panel acts on the top chord (compression chord), you will notice a significant increase in lateral and torsional stiffness.
Alternatively, the lateral stiffness of the truss can be improved by:
Fixing the bottom chords at the supports, or
Increasing the cross-sectional dimensions
However, both options require structural modifications and may only be feasible depending on the project constraints.