You want to take as little off as possible or as mentioned use spacers. Even under daily driving those brackets see immense heat and will warp if too much material is removed.