Linux open big text file

How to open a large text file on Linux

Last updated on October 7, 2020 by Dan Nanni

In the era of «big data», large text files (GB or more) could be commonly encountered around us. Suppose you somehow need to search and edit one of those big text files by hand. Or you could be analyzing multi-GB log files manually for specific troubleshooting purposes. A typical text editor may not be designed to deal with such large text files efficiently, and may simply get choked while attempting to open a big file, due to insufficient memory.

If you are a savvy system admin, you can probably open or touch an arbitrary text file with a combination of cat , tail , grep , sed , awk , etc. In this tutorial, I will discuss more user-friendly ways to open (and possibly edit) a large text file on Linux.

Vim with LargeFile Plugin

Vim text editor boasts of various plugins (or scripts) which can extend Vim’s functionality. One such Vim plugin is LargeFile plugin. This plugin allows you to load and edit large files more quickly by turning off several Vim features such as events, undo, syntax highlighting, etc.

To install the LargeFile plugin on Vim, first make sure that you have Vim installed.

For Ubuntu, Debian or Linux Mint:

For Fedora, CentOS or RHEL:

Then download the LargFile plugin from Vim website. The latest version of the plugin is 5, and it will be saved in Vimball format ( .vba extension).

To install the plugin in your home directory, you can open the .vba file with Vim as follows.

Enter :so % and press ENTER within Vim window to install the plugin in your home directory.

After this, enter :q to quit Vim.

The plugin will be installed at

/.vim/plugin/LargeFile.vim . Now you can start using Vim as usual.

What this plugin does is to turn off events, undo, syntax highlighting, etc. when a large file is loaded on Vim. By default, files bigger than 100MB are considered large by the plugin. To change this setting, you can edit

/.vimrc file (create one if it does not exist).

To change the minimum size of large files to 10MB , add the following entry to

While the LargeFile plugin can help you speed up file loading, Vim itself still cannot handle editing an extremely large file very well, because it tries to load the entire file in memory. For example, when a 1GB file is loaded on Vim, it takes as much memory and swap space, as shown in the top output below.

So if your files are significantly bigger than the physical memory of your Linux system, you can consider other options, as explained below.

glogg Log Explorer

If all you need is read-only access to a text file, and you don’t have to edit it, you can consider glogg , which is a GUI-based standalone log analyzer. The glogg analyzer supports filtered views of an input text file, based on extended regular expressions and wildcards.

To install glogg on Debian (Wheezy and higher), Ubuntu or Linux Mint:

To install glogg on Fedora (17 or higher):

To open a text file with glogg :

glogg can open a large text file pretty fast. It took me around 12 seconds to open a 1GB log file.

You can enter a regular expression in the Text field, and press Search button. It supports case-insensitive search and auto-refresh features. After searching, you will see a filtered view at the bottom window.

Compared to Vim, glogg is much more lightweight after a file is loaded. It was using only 83MB of physical memory after loading a 1GB log file.

JOE Text Editor

JOE is a light-weight terminal based text editor released under GPL. JOE is one of few text editors with large file support, and allows opening and editing files larger than memory.

Besides, JOE supports various powerful text editing features, such as non-destructive editing, search and replace with regular expression, unlimited undo/redo, syntax highlighting, etc.

To install JOE on Ubuntu, Debian or Linux Mint:

To install JOE on Fedora, CentOS or RHEL:

To open a text file for editing, run:

Loading a large file on JOE is a little bit sluggish, compared to glogg above. It took around 30 seconds to load a 1GB file. Still, that’s not too bad, considering that a file is fully editable now. Once a file is loaded, you can start editing a file in terminal mode, which is quite fast.

The memory consumption of JOE is impressive. To load and edit a 1GB text file, it only takes 47MB of physical memory.

If you know any other way to open/edit a large text file on Linux, share your knowledge!

Support Xmodulo

This website is made possible by minimal ads and your gracious donation via PayPal (Credit Card) or Bitcoin ( 1M161JGAkz3oaHNvTiPFjNYkeABox8rb4g ).

Источник

Как открыть большой файл в Linux

В эпоху большого количества информации, большие файлы (размером больше 500 Мб) могут встречаться очень часто. Предположим что вам нужно найти информацию или отредактировать такой файл, но обычные текстовые редакторы не приспособлены для этой задачи и могут просто зависнуть при попытке открыть файл размером несколько гигабайт из-за нехватки памяти.

Если вы опытный системный администратор вы можете открыть такой файл используя комбинацию консольных команд cat, tail, grep, sed, awk и т д. В этой статье я расскажу о более дружелюбных способах открытия и редактирования больших файлов в Linux.

Сначала давайте создадим текстовый файл для экспериментов размером 500 Мб. Для этого воспользуемся командой:

yes «losst.ru » | head -c 500MB > test.txt

1. Большие файлы в Vim

У текстового редактора Vim есть множество различных плагинов или скриптов которые расширяют его функциональность. Одним из таких плагинов есть LargeFile Plugin.

Этот плагин ускоряет загрузку и редактирование больших файлов путем отключения таких дополнительных функций Vim как события, отмена последнего действия, подсветка синтаксиса и т д.

Чтобы установить LargeFile plugin сначала необходимо установить vim, если он еще не установлен. Для Debian, Ubuntu, Linux Mint:

sudo apt install vim

В Fedora, ContOS, или RHEL:

sudo yim install vim-enhanced

Теперь скачайте плагин с официального сайта последняя версия на данный момент 5. Для установки в домашнюю директорию вы можете открыть vba файл расширения с помощью vim:

Наберите :so % и нажмите Enter, чтобы Vim установил расширение в домашнюю папку. Затем можете набирать :q для выхода. Плагин будет установлен в

/.vim/plugin/LargeFile.vim и вы сможете его использовать.

Когда большой файл загружается в vim этот плагин отключает события, отмену действий и подсветку синтаксиса. По умолчанию большими файлами считаются файлы размером более 100 Мб. Чтобы изменить эту настройку нужно отредактировать файл

/.vimrc. Например, будем считать большими файлы больше 10 Мб:

Несмотря на то что плагин LargeFile может ускорить загрузку и просмотр больших файлов сам vim справляется с редактированием больших файлов не очень хорошо потому что он пытается загрузить файл целиком в оперативную память. Например при загрузке моего текстового файла размером 500 Мб утилита будет занимать достаточно много оперативной памяти:

При этом на открытие файла ушло примерно 10-15 секунд. Так что если ваши файлы имеют размер больше чем количество оперативной памяти вам необходимо рассматривать другие варианты.

2. Glogg Log Explorer

Если все что вам нужно это просмотр содержимого файла и вы не собираетесь его редактировать вы можете использовать Glogg. Это анализатор логов с графическим интерфейсом. Glogg поддерживает фильтрацию вывода с помощью расширенных регулярных выражений и масок.

Для установки Glogg в Debilan (Wheezy и выше) Ubuntu и Linux Mint наберите:

sudo apt install glogg

Для установки в Fedora:

sudo yum install glogg

Для открытия текстового файла просто выполните:

Glogg открывает большие файлы очень быстро. Мой тестовый файл открылся практически мгновенно:

Вы можете ввести регулярное выражение в текстовое поле и и нажать поиск. Поддерживается поиск с учетом регистра, результаты поиска вы увидите в нижнем окне.

По сравнению с Vim, Glogg более легкий, для загрузки файла журнала размером 500 Мб было использовано всего лишь 141 Мб оперативной памяти.

3. JOE Text Editor

JOE это легкий консольный текстовый редактор выпущенный под лицензией GPL. JOE один из немногих текстовых редакторов поддерживающих работу с большими файлами. С его помощью можно открывать и редактировать файлы намного большего размера чем размер оперативной памяти.

К тому же JOE поддерживает множество мощных функций для редактирования текста, таких как: безопасное редактирование, поиск и замена по регулярным выражениям, неограниченная отмена и повтор операций, подсветка синтаксиса и т д.

Для установки JOE в Debian, Ubuntu, или Linux Mint наберите:

sudo apt install joe

Для Fedora, CentOS или RHEL:

sudo yum install joe

Чтобы открыть текстовый файл для редактирования выполните:

Загрузка больших файлов в JOE происходит немного медленнее чем в Glogg. На загрузку файла размером 500 Мб ушло примерно 15 секунд. Но это не так уж плохо учитывая что сразу после загрузки файл можно полноценно редактировать.

Потребление памяти JOE впечатляет. На загрузку и редактирование текстового файла размером 500 Мб было использовано 53 Мб физической памяти.

Если вам известный еще какой нибудь способ редактировать большие файлы, сообщите нам в комментариях.

Источник

9 Simple Ways to Effectively use Less Command in Linux

There are ways to read text files in Linux command line. Cat is perhaps the most elementary command that almost every Linux user knows in this regard. But cat is not always the best option for displaying the content of a file.

Imagine that you have a text file with over a thousand line. Using the cat command will simply fill the entire terminal screen with those lines. You cannot perform any more operations like searching for a particular text after you have used the cat command here.

This is where less command helps you big time.

What is the less command in Linux?

With less, you can read large text files without cluttering your terminal screen. You can also search for text and monitor files in real time with it.

Some people prefer using Vim for reading large text files. But less is faster than Vim or other such text editors because it doesn’t read the entire file before starting. Since less is ‘read only‘, you don’t have the risk of accidentally editing the files you are viewing.

The syntax for the less command is extremely simple:

There are numerous options with less command but it’s better to focus on the practical usage that will be more useful for you.

Let’s see some of the most useful examples of less command in Linux.

Practical examples of less command in Linux

Let’s see how to use the less command in Linux with some practical examples.

It is better to work with a big file to understand the usage of the less command. Instead of creating a huge text file by hand, I would advise copying the existing file from /etc/services to your home directory or wherever you want to practice these commands.

/etc/services is a big file with hundreds of line and once copied, you can use it for your practice.

1. View a text file with less

As showed in the syntax, you can use the less command to view a file in the following fashion:

The output will be something like this:

Viewing text file with less

Note: If you view a short text file with less, you would see empty blank lines at the top. Don’t panic. There are no extra lines in your file. It’s just the way less displays them.

2. Exit from less

If you are not used to of less command, you might struggle to find how to exit less. Trust me it’s not at all complicated. Simply press ‘q’ at any given point to exit from less.

I added the existing before so that you may follow the rest of the less command examples easily as you would need to exit the files between different examples (if you are practicing the commands while reading this article).

3. Moving around in less

The output of less is divided into sort of pages. You’ll see only the text that fills up to your terminal screen.

You can use the up and down arrow keys to move line by line. If you want to move page by page, use space key to move to next page and ‘b’ key to go back to the previous page.

If you want to move to the beginning of the file, use ‘g’ key. If you want to go to the end of the file, press ‘G’ key.

• Up arrow – Move one line up
• Down arrow – Move one line down
• Space or PgDn – Move one page down
• b or PgUp – Move one page up
• g – Move to the beginning of the file
• G – Move to the end of the file
• ng – Move to the nth line

4. Display line numbers with less

If you want to see the line numbers in the less command output, you can use the option N in the following manner:

You’ll see an output like this:

Displaying line numbers in less

5. Finding text in less

If you have a large text file, it’s better to search for a specific piece of text rather than reading it line by line in order to find it manually.

To find a word or phrase or even a regex pattern, press / and type whatever you want to find.

As you can see, the matched text is highlighted.

Searching text in less

If there are more than one match, you can move to the next matched text by pressing ‘n’ key. You can move back to the previous match with ‘N’ key.

The ‘/pattern’ performs a forward search. You can use ‘?pattern’ to perform a backward search. Personally, I prefer doing a forward search and then press n or N to cycle through all the matches. No need to worry about a forward or backward search.

By default, search in less is case sensitive. To ignore case, you can use less with -I option

If you forgot to use this option, don’t worry. You can also press ‘-I’ key combination before performing a search inside less.

Tip: If you use &pattern it will display only the lines that matched the pattern.

6. Marking interesting points

While you are scrolling through a big text file and you find something interesting, but you also need to continue checking the file, how would you remember that interesting point? Answer is marking.

Less allows you to add marks, sort of flags, to any line. You can add a mark by pressing the key ‘m’ followed by a letter.

When you want to go back to this mark, simply press ‘ followed by that letter:

You are not limited to a single mark point. You can add multiple marks in less. Simply use a different letter for each marked position.

7. Monitor files in real time with less command

You probably already know that you can use tail command to monitor log files in real time.

You can also do the same with less command. You can see the new lines being added to a file in real time by using the +F option.

It will display the last page of the file and then wait for the new data to be added. Note that you cannot perform the regular moving up and down, back and forth in this mode.

To exit the real time monitoring of log files, press Ctrl+C. With this, you will be back to the normal view of the less command and can exit the file normally.

8. View multiple files with less command

I’ll be honest with you. This is not my favorite less command example but you can totally do this.

To open multiple files with less, simply input the file names one by one:

You’ll see that it lists the file name along with its position in the list of the files.

Viewing multiple files with less

You can view other files in the list using these keys:

• :n – view the next file in the list
• :p – view the previous file in the list

9. Using less command with pipes

The less command can be used in conjugation with other commands using pipes. It is particularly useful when you know that the output of a certain command is going to be huge.

For example, the output of dmesg command is usually in thousands of lines. You don’t want it to flood your screen and you won’t be able to analyze the output as well. Pipe it with less and you’ll have a more friendly way of reading this output.

Bonus Tip: Edit a file with less command in Linux

No, you cannot do that. You cannot edit a file in less.

One of the biggest advantages of less command is that it provides a ‘read-only’ view.

If you cannot edit text with less then why did I add this as a less command example? Because when you feel like you need to edit the file you are viewing, simply press the ‘v’ key.

It will open the file in the default command line text editor of your Linux system. For Ubuntu based system, it should be opened in Nano editor.

There is more with less

I hope you find these less command examples useful while using Linux. Of course, there could be many more usage of less commands. You can explore them by going through all the options of less command.

There is a similar ‘more command‘ that is also popular. I prefer using less because it is more user friendly.

If you have any questions or suggestions, please share it in the comments section below.

Источник