allowing only privileged users to see production branch build artifacts

Is there a way to only allow privileged users (like master or owner only) to see and download production branch build artifacts? We would like to restrict the access to build artifacts with confidential salt and db access passwords etc... However builds of other branches should be fine to see and download or deploy to stage server...