(b) write an expression for the net external torque acting on the system of the two blocks + wheel + string about the wheel's axle. what are its rotation sense and vector direction? [note: the string is part of the system, so the string tension forces provide internal torques, not external torques.]

τext =(M*R*g)*(m₂ - m₁) / (m₁ + m₂)

The wheel is rotating clockwise. Its vector direction is - k  which is perpendicular to the plane of the wheel.

Explanation:

Given info

m₁ = mass of block 1

m₂ = mass of block 2

m₁ < m₂

M = Mass of the wheel

R = Radius of the wheel

I = Moment of inertia of the wheel = M*R²

We assume that the wheel is rotating clockwise since m₁ < m₂

In order to get an expression for the net external torque acting on the system we can apply the following equation

τext = I*α

where α is the angular acceleration of the wheel which can be found as follows

at = R*α   ⇒    α = at / R

then we have to compute the acceleration of the system (at), using the Newton's 2nd Law

Block 1:

∑Fy = m₁*at  (↑+)  ⇒    T - m₁*g = m₁*at    (I)

Block 2:

∑Fy = m₂*at  (↓+)  ⇒    - T + m₂*g = m₂*at    (II)

If  we apply  

(I) + (II)  ⇒        at = g*(m₂ - m₁) / (m₁ + m₂)

Now, we get α:

α = at / R = (g*(m₂ - m₁) / (m₁ + m₂)) / R

⇒     α = (g / R)*(m₂ - m₁) / (m₁ + m₂)

Finally

τext = I*α = (M*R²)*((g / R)*(m₂ - m₁) / (m₁ + m₂))

⇒   τext =(M*R*g)*(m₂ - m₁) / (m₁ + m₂)

The wheel is rotating clockwise. Its vector direction is - k  which is perpendicular to the plane of the wheel.