我正在尝试从某些网址下载图片。对于某些图片,它工作正常,但对于其他图片,我会收到403错误。
例如,这个:http : //blog.zenika.com/themes/Zenika/img/zenika.gif
此图片访问不需要任何身份验证。您可以在链接上单击自己,并使用200状态代码验证该链接可用于您的浏览器。
以下代码产生一个异常:new java.net.URL(url).openStream()。相同的是,org.apache.commons.io.FileUtils.copyURLToFile(new java.net.URL(url), tmp)它们openStream()在引擎盖下使用相同的方法。
new java.net.URL(url).openStream()
org.apache.commons.io.FileUtils.copyURLToFile(new java.net.URL(url), tmp)
openStream()
java.io.IOException: Server returned HTTP response code: 403 for URL: http://blog.zenika.com/themes/Zenika/img/zenika.gif at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626) ~[na:1.7.0_45] at java.net.URL.openStream(URL.java:1037) ~[na:1.7.0_45] at services.impl.DefaultStampleServiceComponent$RemoteImgUrlFilter$class.downloadAsTemporaryFile(DefaultStampleServiceComponent.scala:548) [classes/:na] at services.impl.DefaultStampleServiceComponent$RemoteImgUrlFilter$class.services$impl$DefaultStampleServiceComponent$RemoteImgUrlFilter$$handleImageUrl(DefaultStampleServiceComponent.scala:523) [classes/:na]
我使用Scala / Play Framework开发。我尝试使用内置的AsyncHttpClient。
// TODO it could be better to use itetarees on the GET call becase I think AHC load the whole body in memory WS.url(url).get.flatMap { res => if (res.status >= 200 && res.status < 300) { val bodyStream = res.getAHCResponse.getResponseBodyAsStream val futureFile = TryUtils.tryToFuture(createTemporaryFile(bodyStream)) play.api.Logger.info(s"Successfully downloaded file $filename with status code ${res.status}") futureFile } else { Future.failed(new RuntimeException(s"Download of file $filename returned status code ${res.status}")) } } recover { case NonFatal(e) => throw new RuntimeException(s"Could not downloadAsTemporaryFile url=$url", e) }
使用此AHC代码,它可以正常工作。有人可以解释这种现象,为什么我的URL.openStream()方法出现403错误?
URL.openStream()
如前所述,某些托管服务商使用诸如UserAgent之类的标头来防止这种入侵:
这不起作用:
val urls = """http://blog.zenika.com/themes/Zenika/img/zenika.gif""" val url = new URL(urls) val urlConnection = url.openConnection() val inputStream = urlConnection.getInputStream() val bufferedReader = new BufferedReader(new InputStreamReader(inputStream))
这有效:
val urls = """http://blog.zenika.com/themes/Zenika/img/zenika.gif""" val url = new URL(urls) val urlConnection = url.openConnection() urlConnection.setRequestProperty("User-Agent", """NING/1.0""") val inputStream = urlConnection.getInputStream() val bufferedReader = new BufferedReader(new InputStreamReader(inputStream))