从第三方获得大量数据,怎么高效处理?或者帮我出一个思路
我现在调用第三方的查询接口,查询条件可分页查询(分页必传),我现在是每次查询1000条,直到查询不出来,查询的数据就是这样
{
code:
message:
data: {
"content": [{
#人口信息
"id": "dd8b0806-598a-40da-b0f7-1fad557cd99c",
"name": "王二",
"gender": 0,
"type": 1108,
"imageUrl": ".\\images\\791870be-8fa2-4297-8a51-55f6b05e8a05\\DF5E2EBC-54A0-4F4F-A323-D13E6142E526\\dd8b0806-598a-40da-b0f7-1fad557cd99c.jpg",
"createTime": 1573120249000,
"wxOpenId": null,
"delFlag": true,
"enable": true,
"phone": "00000000000",
"password": "579529",
"communityId": "D04C6714-06F6-4034-8A97-A7102A0ACE2F",
"rooms": [ #房屋信息
{
"id": "DF5E2EBC-54A0-4F4F-A323-D13E6142E526",
"number": "0000",
"name": "0",
"code": null,
"unit": { #单元信息
"id": "791870be-8fa2-4297-8a51-55f6b05e8a05",
"name": "0单元",
"building": { #楼栋信息
"id": "7718190e-42b5-47d4-a35f-c6b39d027d9f",
"name": "物业办公室",
"community": { #小区信息
"id": "D04C6714-06F6-4034-8A97-A7102A0ACE2F",
"name": "鸿翔逸境小区",
"key": "1234567890",
"dataSourceType": 3
},
"hibernateLazyInitializer": {}
},
"hibernateLazyInitializer": {}
},
"delFlag": false
}]
}],
"pageable": {
"sort": {
"sorted": true,
"unsorted": false,
"empty": false
},
"offset": 0,
"pageSize": 20,
"pageNumber": 0,
"unpaged": false,
"paged": true
},
"totalPages": 1,
"last": true,
"totalElements": 4,
"number": 0,
"size": 20,
"sort": {
"sorted": true,
"unsorted": false,
"empty": false
},
"numberOfElements": 4,
"first": true,
"empty": false
}
}
我想先把数据落库,然后数据再解析,把人口信息放到人口表里,把房屋信息放到房屋表里,小区信息放到小区表里。
我现在的问题
1.在不影响其他功能的情况下执行存储
2.高效的执行,时间短,项目不会蹦
我的思路是想把数据先存到redis中,然后从redis一条条解析录入,但是具体怎么怎么放到redis,又怎么取,没思路
各位大哥,有没有别的思路,帮我梳理梳理,谢谢