{{announcement.body}}
{{announcement.title}}

Using Nexus OSS Repository in An Isolated Environment

DZone 's Guide to

Using Nexus OSS Repository in An Isolated Environment

What happens if you want to access and work with open source libraries but don't have internet? Here's an isolated Nexus solution.

· Cloud Zone ·
Free Resource

No Internet Access, No Libraries?

In our workplace, like in many other big firms, government offices, and banks, the internal network is isolated from the outside world. Nothing goes out and nothing goes in. But, how can you conveniently work with open source libraries if you cannot use Maven or npm (and other repositories) to fetch needed libraries?

Nexus, and other similar products can be installed in pairs, to bridge between networks. But as far as I know, that topology can only work in synchronous mode and you cannot add your own asynchronous checks like using sandboxes or anti-virus checks — and that can be a problem. Plus, we wanted to use only the free, open-source version of Nexus (OSS), as we try to rely on open source as much as possible.

So we came up with a solution for that, drawn here:


Nexus asynchronous network

Nexus asynchronous network


Basically, we act as a proxy to the internal Nexus, receiving its requests (initiated by developers issuing  mvn install or  npm install  commands). Next, we pass the requests to the external network via IBM’s Data Power (which verifies the request, in case someone tries to exploit the endpoint — similar products exist as well). Data Power passes the request to an external proxy, with internet access. This external proxy downloads the required library, along with all its dependencies, and passes it on to the organizational file checking system (sandboxes, antivirus, etc.). If the file is approved, it is copied into the internal network (via a vault or a similar mechanism). There, the internal proxy comes into live again, picks up those libraries and uploads them into Nexus. Then the developer can retry to fetch the libraries and...  voilà! The libraries are there and he can use them in his code.

Inner Proxy

The inner proxy is a Java Spring Boot application. It has two parts:

  • The first part is a REST controller, exposing a GET URL that accepts requests from Nexus. Using a servlet filter these requests are caught and sent to the DataPower (Listing 1).

  • The second part watches a designated folder, to which the approved libraries are copied (wrapped in a zip file). It extracts the libraries from the zip and uploads them into Nexus (Listing 2).

Outer Proxy 

The outer proxy is also a Java Spring Boot application. It accepts GET requests for the needed libraries. Then it downloads the libraries and their dependencies from Maven or npm, and places them in a designated folder. The organizational file checking system will take the files from there, check them, and if they are approved they will be moved into the inner network wrapped as a zip file (Listing 3).

That’s it! You are welcome to use a similar solution in your environment and start enjoying the wealth of open source libraries out there…

Listing 1 - Intercepting the Requests

Java
 




x
125


 
1
public class NexusInterceptingFilter extends DispatcherServlet {
2
 
          
3
 
          
4
 
          
5
    private static final long serialVersionUID = 1L;
6
 
          
7
    private static final String DP_URL_KEY = "dp_url";
8
 
          
9
    private static final String SEPARATOR = "/";
10
 
          
11
    protected static Logger logger = LoggerFactory.getLogger(NexusInterceptingFilter.class);
12
 
          
13
 
          
14
 
          
15
    @Override
16
 
          
17
    protected void doDispatch(HttpServletRequest request, HttpServletResponse response) throws Exception {
18
 
          
19
        if (!(request instanceof ContentCachingRequestWrapper)) {
20
 
          
21
            request = new ContentCachingRequestWrapper(request);
22
 
          
23
        }
24
 
          
25
        if (!(response instanceof ContentCachingResponseWrapper)) {
26
 
          
27
            response = new ContentCachingResponseWrapper(response);
28
 
          
29
        }
30
 
          
31
        HandlerExecutionChain handler = getHandler(request);
32
 
          
33
        try {
34
 
          
35
            super.doDispatch(request, response);
36
 
          
37
        } finally {
38
 
          
39
            handleRequests(request, response, handler);
40
 
          
41
            updateResponse(response);
42
 
          
43
        }
44
 
          
45
    }
46
 
          
47
 
          
48
 
          
49
    private void handleRequests(HttpServletRequest requestToCache, HttpServletResponse responseToCache,
50
 
          
51
            HandlerExecutionChain handler) {
52
 
          
53
        logger.info(">>>>> Fetching: "+ requestToCache.getRequestURI());
54
 
          
55
        // This is for java only
56
 
          
57
        if (requestToCache.getRequestURI().endsWith(".pom")) {
58
 
          
59
            // make the request for the lib ( etc. abbot/abbot/0.13.0/abbot-0.13.0.pom )
60
 
          
61
            RestTemplate rest = new RestTemplate();
62
 
          
63
            Environment env = (Environment)ApplicationContextProvider.getBeanByClass(Environment.class);
64
 
          
65
            String dpUrl = env.getProperty(DP_URL_KEY);
66
 
          
67
            String theUrl = dpUrl+"?libName="+getLibRequest(requestToCache.getRequestURI());
68
 
          
69
            logger.info("The request: "+theUrl);
70
 
          
71
            rest.getForObject(theUrl, Void.class);
72
 
          
73
            
74
 
          
75
        }
76
 
          
77
    }
78
 
          
79
 
          
80
 
          
81
    
82
 
          
83
    private String getLibRequest(String requestURI) {
84
 
          
85
        if (requestURI.endsWith(".pom")) {
86
 
          
87
            String[] parts = requestURI.split(SEPARATOR);
88
 
          
89
            String version = parts[parts.length - 2];
90
 
          
91
            String artifactId = parts[parts.length - 3];
92
 
          
93
            StringBuilder grp = new StringBuilder();
94
 
          
95
            for(int i=2;i<parts.length - 3;i++) {
96
 
          
97
                grp.append(parts[i]).append(".");
98
 
          
99
            }
100
 
          
101
            return grp.toString().substring(0,grp.length()-1)+SEPARATOR+
102
 
          
103
                    artifactId+SEPARATOR+version;
104
 
          
105
        }
106
 
          
107
        return null;
108
 
          
109
    }
110
 
          
111
 
          
112
 
          
113
    private void updateResponse(HttpServletResponse response) throws IOException {
114
 
          
115
        ContentCachingResponseWrapper responseWrapper = WebUtils.getNativeResponse(response,
116
 
          
117
                ContentCachingResponseWrapper.class);
118
 
          
119
        responseWrapper.copyBodyToResponse();
120
 
          
121
    }
122
 
          
123
 
          
124
 
          
125
}




Listing 2 - Upload to Nexus

Java
 




xxxxxxxxxx
1
187


 
1
@Component
2
 
          
3
public class ScheduledLibrariesReader {
4
 
          
5
 
          
6
 
          
7
    private static final String MANIFEST = "manifest.txt";
8
 
          
9
    @Value("${vaultFolder}")
10
 
          
11
    private String vaultFolder;
12
 
          
13
    @Value("${tempFolde}")
14
 
          
15
    private String tempFolder;
16
 
          
17
    @Value("${nexus_url}")
18
 
          
19
    private String nexusUploadUrl;
20
 
          
21
 
          
22
 
          
23
    protected static Logger logger = LoggerFactory.getLogger(ScheduledLibrariesReader.class);
24
 
          
25
    
26
 
          
27
    @Scheduled(fixedRate = 60000)
28
 
          
29
    public void readFiles() throws Exception {
30
 
          
31
        // Read all files (= zip for java)
32
 
          
33
        try (Stream<Path> paths = Files.walk(Paths.get(vaultFolder))) {
34
 
          
35
            paths.filter(Files::isRegularFile).forEach(f -> {
36
 
          
37
                try {
38
 
          
39
                    // open the zip to a temp folder we create
40
 
          
41
                    String theFolder = parseAndRename(f);
42
 
          
43
                    // upload all files from this dir, according to the manifest
44
 
          
45
                    upload(theFolder);
46
 
          
47
                    // remove zip and files
48
 
          
49
                    removeAll(theFolder, f);
50
 
          
51
                } catch (Exception e) {
52
 
          
53
                    throw new RuntimeException(e);
54
 
          
55
                }
56
 
          
57
            });
58
 
          
59
        }
60
 
          
61
    }
62
 
          
63
 
          
64
 
          
65
    private String parseAndRename(Path zipFile) throws Exception {
66
 
          
67
        // output dir is under the temp folder in a new folder, based on the zip name
68
 
          
69
        String dirName = tempFolder + "/"
70
 
          
71
                + zipFile.getFileName().toString().substring(0, zipFile.getFileName().toString().lastIndexOf("."));
72
 
          
73
        File destDir = new File(dirName);
74
 
          
75
        if (!destDir.exists()) {
76
 
          
77
            destDir.mkdir();
78
 
          
79
        }
80
 
          
81
        byte[] buffer = new byte[1024];
82
 
          
83
        ZipInputStream zis = new ZipInputStream(new FileInputStream(zipFile.toFile()));
84
 
          
85
        ZipEntry zipEntry = zis.getNextEntry();
86
 
          
87
        while (zipEntry != null) {
88
 
          
89
            File newFile = new File(destDir, zipEntry);
90
 
          
91
            FileOutputStream fos = new FileOutputStream(newFile);
92
 
          
93
            int len;
94
 
          
95
            while ((len = zis.read(buffer)) > 0) {
96
 
          
97
                fos.write(buffer, 0, len);
98
 
          
99
            }
100
 
          
101
            fos.close();
102
 
          
103
            zipEntry = zis.getNextEntry();
104
 
          
105
        }
106
 
          
107
        zis.closeEntry();
108
 
          
109
        zis.close();        
110
 
          
111
        //
112
 
          
113
        return dirName;
114
 
          
115
    }
116
 
          
117
 
          
118
 
          
119
    @SuppressWarnings("unchecked")
120
 
          
121
    private void upload(String theFolder) throws Exception {
122
 
          
123
        List<String> lines = Files.readAllLines(new File(theFolder+"/"+MANIFEST).toPath());
124
 
          
125
        for (String line : lines) {
126
 
          
127
            String[] parts = line.split(":"); // group/artifact/version//filename
128
 
          
129
            ByteArrayOutputStream stdout = new ByteArrayOutputStream();
130
 
          
131
            ByteArrayOutputStream stderr = new ByteArrayOutputStream();        
132
 
          
133
 
          
134
 
          
135
            Map map = new HashMap();
136
 
          
137
            map.put("file", new File(theFolder + "/" + parts[3]));
138
 
          
139
            CommandLine cmdLine = new CommandLine(MAVEN_LOC+"mvn.cmd");
140
 
          
141
            cmdLine.addArgument("deploy:deploy-file");
142
 
          
143
            cmdLine.addArgument("-DgroupId=" + parts[0]);
144
 
          
145
            cmdLine.addArgument("-DartifactId=" + parts[1]);
146
 
          
147
            cmdLine.addArgument("-Dversion=" + parts[2]);
148
 
          
149
            cmdLine.addArgument("-DgeneratePom=true");
150
 
          
151
            cmdLine.addArgument("-Dpackaging=jar");
152
 
          
153
            cmdLine.addArgument("-DrepositoryId=nexus");
154
 
          
155
            cmdLine.addArgument("-Durl="+nexusUploadUrl);
156
 
          
157
            cmdLine.addArgument("-Dfile=${file}");
158
 
          
159
            cmdLine.setSubstitutionMap(map);
160
 
          
161
            DefaultExecutor executor = new DefaultExecutor();                  
162
 
          
163
            PumpStreamHandler streamHandler = new PumpStreamHandler(stdout, stderr);
164
 
          
165
            executor.setStreamHandler(streamHandler);
166
 
          
167
            DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();
168
 
          
169
            executor.execute(cmdLine, resultHandler);
170
 
          
171
            resultHandler.waitFor();
172
 
          
173
            logger.info("Exit code for uploading "+line+" is "+resultHandler.getExitValue());
174
 
          
175
            logger.info("Error is:\n"+stderr.toString());
176
 
          
177
            logger.info("Output is:\n"+stdout.toString());
178
 
          
179
        
180
 
          
181
        }
182
 
          
183
    }
184
 
          
185
 
          
186
 
          
187
    



Listing 3 - Downloading from Maven

Java
 




xxxxxxxxxx
1
145


 
1
@RestController
2
 
          
3
public class ProxyController {
4
 
          
5
 
          
6
 
          
7
    private static final String MANIFEST = "manifest.txt";
8
 
          
9
    private static final String SEPARATOR = "\\";
10
 
          
11
    @Value("${vaultFolder:d:/Temp/repo}")
12
 
          
13
    private String vaultFolder;
14
 
          
15
    protected static Logger logger = LoggerFactory.getLogger(ProxyController.class);
16
 
          
17
    
18
 
          
19
    @GetMapping("/fetch")
20
 
          
21
    // Expecting groupId/artifactId/version
22
 
          
23
    public void fetchLib(@RequestParam String libName) throws Exception {        
24
 
          
25
        // Currently supporting Maven and NPM
26
 
          
27
        if (isMavenLib(libName)) {
28
 
          
29
            List<Path> libs = fetchMavenLibs(libName);
30
 
          
31
            moveToVault(libs);
32
 
          
33
        } else {
34
 
          
35
            // assume npm
36
 
          
37
            fetchJsonLib(libName);
38
 
          
39
        }
40
 
          
41
    }
42
 
          
43
 
          
44
 
          
45
    private void moveToVault(List<Path> libs) throws IOException {
46
 
          
47
        StringBuilder manifest = new StringBuilder();
48
 
          
49
        String zipFile = vaultFolder + "/nexus_" + System.currentTimeMillis() + ".zip";
50
 
          
51
        byte[] buffer = new byte[1024];
52
 
          
53
        ZipOutputStream zos = null;
54
 
          
55
        FileInputStream fis = null;
56
 
          
57
        try {
58
 
          
59
            FileOutputStream fos = new FileOutputStream(zipFile);
60
 
          
61
            zos = new ZipOutputStream(fos);
62
 
          
63
            //
64
 
          
65
            for (Path path : libs) {
66
 
          
67
                manifest.append(getPartsString(path)).append("\n");                
68
 
          
69
                // add to zip
70
 
          
71
                fis = new FileInputStream(path.toFile());
72
 
          
73
                // begin writing a new ZIP entry, positions the stream to the start of the entry data
74
 
          
75
                zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
76
 
          
77
                int length;
78
 
          
79
                while ((length = fis.read(buffer)) > 0) {
80
 
          
81
                    zos.write(buffer, 0, length);
82
 
          
83
                }
84
 
          
85
                zos.closeEntry();
86
 
          
87
                // close the InputStream
88
 
          
89
                fis.close();
90
 
          
91
                //Files.move(p, new File(vaultFolder + "/" + p.getFileName()).toPath());
92
 
          
93
            }
94
 
          
95
            // add the manifest to the zip
96
 
          
97
            zos.putNextEntry(new ZipEntry(MANIFEST));
98
 
          
99
            zos.write(manifest.toString().getBytes());
100
 
          
101
            zos.closeEntry();
102
 
          
103
            //
104
 
          
105
        } finally {
106
 
          
107
            zos.close();
108
 
          
109
        }
110
 
          
111
 
          
112
 
          
113
    }
114
 
          
115
 
          
116
 
          
117
    private List<Path> fetchMavenLibs(String libName) {
118
 
          
119
        logger.info("Fetching lib " + libName);
120
 
          
121
        String groupId = getGroupId(libName);
122
 
          
123
        String artifactId = getArtifactId(libName);
124
 
          
125
        String version = getVersion(libName);
126
 
          
127
        JkDependencySet deps = JkDependencySet.of().and(groupId + ":" + artifactId + ":" + version)
128
 
          
129
                //   e.g. "com.threerings:tripleplay:1.4")
130
 
          
131
                .withDefaultScopes(COMPILE_AND_RUNTIME);
132
 
          
133
        JkDependencyResolver resolver = JkDependencyResolver.of(JkRepo.ofMavenCentral());
134
 
          
135
        List<Path> libs = resolver.resolve(deps, RUNTIME).getFiles().getEntries();
136
 
          
137
        logger.info("Downloaded lobs: {}", libs);
138
 
          
139
        return libs;
140
 
          
141
    }
142
 
          
143
 
          
144
 
          
145
}



Topics:
isolated development ,java ,nexus ,nexus 3 ,open source ,spring boot

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}