我使用io.Copy()复制文件,大约700Mb,但这会导致内存不足
bodyBuf := &bytes.Buffer{} bodyWriter := multipart.NewWriter(bodyBuf) //key step fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) if err != nil { return nil, err } file, err := os.Open(fileName) //the file size is about 700Mb if err != nil { return nil, err } defer file.Close() //iocopy copyLen, err := io.Copy(fileWriter, file) // this cause out of memory if err != nil { fmt.Println("io.copy(): ", err) return nil, err }
错误信息如下:
runtime: memory allocated by OS (0x752cf000) not in usable range [0x18700000,0x98700000) runtime: out of memory: cannot allocate 1080229888-byte block (1081212928 in use) fatal error: out of memory
我为buf分配了足够的内存,这导致bodyWriter.CreateFormFile()中的内存不足
buf := make([]byte, 766509056) bodyBuf := bytes.NewBuffer(buf) bodyWriter := multipart.NewWriter(bodyBuf) fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) // out of memory if err != nil { return nil, err }
这是因为您正在“复制”到bodyBuf,这是内存中的缓冲区,从而迫使Go尝试分配与整个文件一样大的内存块。
bodyBuf
根据您的使用情况,multipart您似乎正在尝试通过HTTP流式传输文件?在这种情况下,请勿将传递bytes.Buffer给multipart.NewWriter,而是直接传递您的http连接。
multipart
bytes.Buffer
multipart.NewWriter