Can Hoa Govern Town's Right Of Way

Location: Raleigh, NC

Our neighborhood has a stormwater pond that the Town inspects on a regular basis to be up to code. I do not have access to this agreement between our neighborhood and the Town. We took out some shrubs from the front of our house and spread the dirt we took out to cover some eroded areas around the stormwater pond. The amount of dirt we spread was about a 10 gallon bucket. Now I have received this email from the HOA management company.

"you have disposed fill dirt/clay/gravel in the road right of way across the street from 100 ABC St. This needs to be removed by Saturday at 12:00pm. If this is not taken care of in a timely fashion, the Board will have it removed at your expense, and it will be billed to your account."

My question is: Does the HOA have a right to govern Town's right of way? The bylaws and covenants do not mention anything about stormwater pond but our general rules document says this - "No activity is permitted on the property which might cause damage to lawns, landscaping, buildings, pavement or other personal property. If damage is caused to any community area or another owner's property due to actions of a unit owner, a household pet, guest or occupant, the owner of the unit will be responsible for repair charges as determined by the Board of Directors and/or the Management Company.".

We did not have any malicious intent and will remove the dirt if we should but want to know if this is even legal on the HOA's part. They have been targeting and harassing us for a while so I want to see if I can add this to the list of grievances before sending them a stern letter about their behavior in the past 4 years.

Edit: I am unable to attach a pic to this post or in comments. If someone knows how, please let me know.